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FOREWORD 


The Software Engineering Laboratory (SEL) is an organization 
sponsored by the National Aeronautics and Space Administra- 
tion, Goddard Space Flight Center (NASA/GSFC) and created 
for the purpose of investigating the effectiveness of soft- 
ware engineering technologies when applied to the develop- 
ment of applications software. The SEL was created in 1977 
and has three primary organizational members: 

NASA/GSFC (Systems Development and Analysis Branch) 

The University of Maryland (Computer Sciences Department) 
Computer Sciences Corporation (Flight Systems Operation) 

The goals of the SEL are (1) to understand the software de- 
velopment process in the GSFC environment; (2) to measure 
the effect of various methodologies, tools, and models on 
this process; and (3) to identify and then to apply success- 
ful development practices. The activities, findings, and 
recommendations of the SEL are recorded in the Software En- 
gineering Laboratory Series, a continuing series of reports 
that includes:? this document. A version of this document was 
also issued as Computer Sciences Corporation document 
CSC/SD-82/6209 . 

The primary contributor to this document is 

David Card (Computer Sciences Corporation) 

Single copies of this document can be obtained by writing to 

Frank E. McGarry 
Code 582,1 
NASA/GSFC 

Greenbelt, Maryland 20771 
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ABSTRACT 


This document is an annotated bibliography of technical 
papers, documents, and memorandums produced by or related to 
the Software Engineering Laboratory (SEL) . More than 
75 publications are summarized. An index of these publica- 
tions by subject is also included. These publications cover 
many areas of software engineering and range from research 
reports to software documentation. 
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SECTION 1 - INTRODUCTION 


This document is an annotated bibliography of technical 
papers, documents, articles, and memoranda produced by or 
related to the Software Engineering Laboratory (SEL) . It is 
intended to provide a quick reference to the published re- 
sults of SEL research and development activities. 

More than 75 publications are summarized in this document. 
Each summary includes the size of the publication (number of 
pages), a description (abstract) of its contents, and its 
original citation. Previous versions and subsequent re- 
printings are also identified where appropriate. 

The publications described here cover many areas of software 
engineering and range from research reports to software doc- 
umentation. They are divided into two groups. Section 2 
describes the major documents produced by the SEL and is 
«,u t’tnged according to the order in which the documents were 
published by the SEL (document number) . Section 3 describes 
technical memoranda and articles produced by the SEL or re- 
lated to SEL activities and is arranged alphabetically by 
author. An index of these publications by subject is in- 
cluded at the end of this document to assist in identifying 
materials related to specific topics. A list of the ab- 
stract numbers of related publications follows each subject 
heading in the index. 

Copies of individual publications listed in this bibliog- 
raphy can be obtained from one or more of the sources shown 
in Table 1-1. The acronyms defined in the table appear 
after each abstract and indicate the document's availabil- 
ity. Any material not labeled with one of these acronyms 
can be obtained only from the author (s). 



Table 1-1. Availability of SEL Literature 


Acronym 


Source 


Address 


FEM 


NSTF 


NTIS 

JAO 


P. E. McGarry 


NASA Scientific and 
Technical Installation 
Facility- 1 - (and source 
above) 

National Technological 
Information Service 2 
(and sources above) 

Journals and other 
private publishers 


Code 582.1 

Bldg. 23 , Room E-231 
Goddard Space Plight 
Center 

Greenbelt, Maryland 
20771 

P. 0. Box 8757 

BWI Airport, Maryland 

21240 


5285 Port Royal Road 
Springfield, Virginia 
22161 

See specific citation 


"^Open to Federal Government agencies only at no charge. 
2 . 

Tnere is a per-page charge for reprinting documents. 
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SECTION 2 - SEL-ORIGINATED DOCUMENTS 
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2.1 PROCEEDINGS FROM THE FIRST SUMMER SOFTWARE ENGINEERING 
WORKSHOP # SEL-76-001 # AUGUST 1976, 194 PAGES 


This document reproduces the presentations made by partici- 
pants at the First Summer Software Engineering Workshop held 
on August 5, 1976, at Goddard Space Flight Center (GSFC) . 

The general topic of the conference was software design. 

The presentations were grouped into the following panels: 

• Requirements analysis and design methodologies 

• Program design languages 

• Automated software tools 

This first workshop surveyed available state-of-the-art 
software development techniques. The specific applicability 
of the techniques to the GSFC environment was also con- 
sidered. Approximately 25 persons attended this meeting. 

NT IS 
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2.2 THE SOFTWARE ENGINEERING LABORATORY ! SEL-77-001, 

V. R. BASIL! r M. V. ZELKOWITZ, F. E. Me GARRY , ET AL. , 
MAY 1977, 1X2 PAGES 

This document is the initial statement of SEL objectives and 
procedures. The basic objective of the SEL is to improve 
the quality of the software development process and its 
products in one specific environment. To do this, it is 
necessary to first establish a baseline for comparison by 
understanding the current process. 

This document discusses the steps to be taken to establish 
this baseline, including software development factors to be 
studied, the types of experiments to be performed, and the 
methods of data collection to be used. The current (as of 
publication) status of research and development activities 
is also reviewed. The appendixes contain the following 
materials: 

• Data collection forms 

• Data base file formats 

• Summaries of software projects studied 

This document was also issued as University of Maryland 
Technical Report TR-535. NSTF 
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2.3 PROCEEDINGS FROM THE SECOND SUMMER SOFTWARE ENGINEERING 
WORKSHOP , SEL-77-002, SEPTEMBER 1977, 146 PAGES 

This document reproduces the presentations made by partici- 
pants at the Second Summer Software Engineering Workshop 
held on September 19, 1977, at GSFC. The presentations were 
grouped into the following panels: 

• Experimental design 

• Models, measures, and metrics 

• Data collection 

• Software engineering experiences 

This second workshop attempted to communicate with the 
larger software engineering research community. Approaches 
and experiences with the design of experiments and data col- 
lection were reviewed. Approximately 55 persons attended 
this meeting. NTIS 



2.4 STRUCTURED FORTRAN PREPROCESSOR (SFORT) , SEL-77-003, 

B. CHU AND D. S. WILSON, SEPTEMBER 1977, 51 PAGES 

This document describes the Structured FORTRAN Preprocessor 
(SFORT) used in the development of flight dynamics software. 
The use of SFORT extends the capabilities of the standard 
FORTRAN language and enables a user to write structured, 
top-down, label-free, FORTRAN-like code. SFORT converts 
this extended FORTRAN into standard FORTRAN, which can then 
be used as input to a standard FORTRAN compiler. 

This document discusses the six additional statements that 
may be employed in conjunction with the standard FORTRAN 
statements. The additional statements are 

• DOWHILE (FORTRAN logical expression) 

• ENDDO 

• .IF (FORTRAN logical expression) 

• ELSE 

• ENDIF 

• INCLUDE member name 

Examples of the use of these statements in coding, as well 
as the IBM S/360 job control language (JCL) necessary to 
execute the preprocessor, are included. This document was 
also issued as Computer Sciences Corporation document 
CSC/TM- 77/625 6. FEM 
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2. 5 GSFC NAVPAK DESIGN SPECIFICATION LANGUAGES STUDY , 
SEL-77-004, P. A. SCHEFFER AND C. E. VELEZ, 

OCTOBER 1977, 37 PAGES 

This document reports the results of an analysis of the ap- 
plicability of software specification and design languages 
to the development of flight dynamics software at GSFC. Two 
language systems were studied: HOS/AXES and PSL/PSA. The 

approach was to develop software designs, using each of 
these language systems, from the requirements of an already- 
implemented system. The results of the two methodologies 
were then compared with each other and with the actual de- 
sign implemented. This document was also issued as a Martin 
Marietta Corporation technical report. NSTF 
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2.6 A DEMONSTRATION OF AXES FOR NAVPAK , SEL-77-005, 

M. HAMILTON AND S. ZELDIN, SEPTEMBER 1977, 84 PAGES 

This document describes the results of a demonstration proj 
ect employing a system specification language tool to re- 
spond to a flight dynamics software development requirement 
The tool, AXES, was developed by Higher Order Software, Inc 
Specific areas were selected for the demonstration from the 
NAVPAK software requirements. The complexity of these re- 
quirements and the effort required to obtain a full under- 
standing of them prevented the development of a complete 
design within the demonstration period. This document de- 
scribes the following aspects of the demonstration project: 

• Principles of HOS/AXES 

• Description of NAVPAK 

• HOS/AXES representation of NAVPAK specifications 

The appendixes contain some unrelated examples and other 
background material. This document was also issued as 
Higher Order Software, Inc., Technical Report 9. 
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2.7 FORTRAN STATIC SOURCE CODE ANALYZER DESIGN AND MODULE 

DEiSckiPTlONSj , SEL-7d-00i, E. M. 0 1 NEILL# S. R. WALIGORA, 
AND C. E. GOOREVICH, FEBRUARY 1978, 98 PAGES 

This document contains the system design and individual mod- 
ule descriptions of the FORTRAN Static Source Code Analyzer 
Program (SAP) . This program is a source code scanning and 
statistical reporting program whose purpose is to collect 
data from actual software for studies of software measures 
by the SEL. The measures extracted by SAP. include counts of 
the number of 

• Source lines 

• Comment lines 

• Executable lines 

• Decisions 

• External references 

These statistics are reported on a module-by-module basis 
and as final totals. This document was also issued as Com- 
puter Sciences Corporation document CSC/TM-78/6012. FEM 


2.8 FORTRAN STATIC SOURCE CODE ANALYZER PROGRAM (SAP) USER'S 
GUIDE (REVISION 1) , SEL-78-102, W. J. DECKER AND 
W. A. TAYLOR, SEPTEMBER 1982, 133 PAGES 

This document presents the FORTRAN Static Source Code Ana- 
lyzer Program (SAP) User's Guide (Revision 1). SAP is a 
software tool designed to assist SEL personnel in conducting 
studies of software measures. SAP scans FORTRAN source code 
and produces reports that present statistics and measures of 
statements and structures that make up a module. 

Revision 1 is the result of program modifications to provide 
several new reports, additional complexity analysis, and 
recognition of all statements described in the FORTRAN 77 
standard. It provides instructions for operating SAP and 
contains information useful in interpreting SAP output. The 
appendixes contain descriptions of the derivation of 
Halstead's and McCabe's measures. This document was also 
issued as Computer Sciences Corporation document 
CSC/S D- 8 2/60 44. NSTF 


The previous version of this document was FORTRAN Static 
Source Code Analyzer Program (SAP) User's Guide , SEL-78-0Q2, 
E. M. O'Neill, S. R. Waligora, C. E. Goorevich, et al., 
February 1978. 
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2.9 EVALUATION OP DRAPER NAVPAK SOFTWARE DESIGN, SEL-78-003, 
K. TA&Akl AMD 'f. fi. McOARRif, jONfi"T9'7T,"''2T“PAGES 

This document describes the results of an evaluation of a 
software design demonstration project. The design methodol- 
ogy was developed by the Charles Stark Draper Laboratories 
(CSDL) . A sample flight dynamics project was designed by 
CSDL from the NAVPAK software requirements using this meth- 
odology. The end product was evaluated to determine the 
effectiveness of the CSDL methodology. FEM 
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2.10 STRUCTURED FORTRAN PREPROCESSOR (SFQRT) PDP-11/70 
User's 6uidb , sel-78-004, d. s. wilson and b. chu, 
SEPTEMBER 1978, 19 PAGES 

This document provides users of the Structured FORTRAN Pre- 
processor (SFORT) 11/70 with the information needed to write 
programs using SFORT constructs. SFORT extends the capabil- 
ities of the FORTRAN IV language and enables a user to write 
structured , top-down, label-free, FORTRAN-like code. SFORT 
converts this extended FORTRAN into standard FORTRAN, which 
can then be used as input to a compiler. The additional 
statements processed by SFORT include 

• DOWHILE (FORTRAN logical expression) 

• ENDDO 

9 .IF (FORTRAN logical expression) 

9 ELSE 

• ENDIF 

• INCLUDE member name 

This document was also issued as Computer Sciences Corpora- 
tion document CSC/SD-78/6128. NSTF 
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2.11 PROCEEDINGS from the third summer software engineering 
WORKSHOP , SEL-78-005, SEPTEMBER 1978, 132 PAGES 


This document reproduces the presentations made by partici- 
pants at the Third Summer Software Engineering Workshop held 
on September 18, 1978, at GSFC. The presentations were 
grouped into the following panels: 

• The data collection process 

• Validation of software development models 

• Measuring software development methodologies 

• Current activities and future directions 

Many of the discussions at this third workshop dealt with 
"how" one collects software data and "how” one conducts 
successful software experiments. Approximately 70 persons 
attended this meeting. NTIS 


2-12 


8978 


2,12 GSFC SOFTWARE ENGINEERING RESEARCH REQUIREMENTS 
ANALYSIS StfUDY , £ EL-1 6 -'00 67 P .A. SCHEFFER AND 
C. E. VELEZ , NOVEMBER 1978, 26 PAGES 

This document reports the results of a study of the applica- 
bility of requirements languages to flight dynamics software 
development at GSFC, The specific objectives of the study, 
which are explained in this document, were to 

• Determine the impact of requirements language use 
on software design 

• Demonstrate the application of a requirements lan- 
guage on a flight dynamics development problem 

• Evaluate the utility of the Multi-Level Expression 
Design Language - Requirement Level (MEDL-R) in the 
GSFC environment 

• Determine the desirable characteristics of a re- 
quirements language tool for use in the GSFC en- 
vironment 

This document was also issued as a Martin Marietta Corpora- 
tion technical memorandum. NSTF 
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2.13 APPLICABILITY OF THE RAYLEIGH CURVE TO THE SEL ENVIRON * 
MBNT , SEL-?8-607r T. E. MAPP, DEcjEMBfeR 1976 # 2 7 PAGES 

This document reviews the resource utilization model Cor 
software development# which is based on the Rayleigh curve 
developed by Norden and Putnam. A Rayleigh curve is fitted 
to data provided by the SEL. Parabolas# trapezoids# and 
straight lines are also fitted to the same data. The parab- 
ola and trapezoid give about as good a fit as the Rayleigh 
curve. Therefore# this document concludes that while the 
Rayleigh curve may be an appropriate model for resource ex- 
penditures, it is not necessarily the best model for small- 
to medium-size projects. NSTF 
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2.14 SIMPL-D DATA BASE REFERENCE MANUAL . SEL-79-001, 

M. V. ZELKOWITZ, JULY 1979, 53 PAGES 

This document describes a data base system for software en- 
gineering data implemented on the Univac 1100/42 computer at 
the University of Maryland. The data base contains flight 
dynamics development data collected by the SEL. The soft- 
ware is implemented in the SIMPL-D language. Data base 
files are sequentially organized. The document contains the 
following sections: 

• Overview of the application area 

• Description of the data base organization 

• User's guide to data base functions 

• Implementation considerations for the 1100/42 

• Programmer's guide to data base maintenance 

This document was also issued as University of Maryland 
Technical Report TR-788. PEM 
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2.15 THE SOFTWARE ENGINEERING LABORATORY; RELATIONSHIP 

EQUATIONS, SEL-79-002, K. FREBURGER AND V. R. BASXLI, 
MAY 197?; 67 PAGES 

This document presents the results of an analysis of several 
factors affecting software development. The analysis was 
based on data collected by the SEL. Relationships among the 
following measures were studied: 

• Total effort (staff-months) 

• Lines of delivered code (thousands) 

• Lines of developed code (thousands) 

• Percentage of developed code 

• Number of modules 

• Number of developed modules 

• Percentage of developed modules 

• Project duration (months) 

• Pages of documentation 

• Productivity 

• Average staff size 

Estimating equations were derived from the measures by sta- 
tistical analysis and were then compared with results 
obtained by Walston and Felix in a similar study. This doc- 
ument was also issued as University of Maryland Technical 
Report TR-764. NSTF 


2.16 COMMON SOFTWARE MODULE REPOSITORY (CSMR) SYSTEM DE - 
SCRIPTION AND USER'S GUIDE , SEL-79-003, ” 

C. E. GOOREVICH, A. L. GREEN , AND S. R. WALIGORA, 
AUGUST 1979, 156 PAGES 

This document is the system description and user's guide for 
the Common Software Module Repository (CSMR) . The CSMR pro- 
gram is a software library utility that provides interactive 
access to a data base of software modules. 

This document describes library control procedures, program 
capabilities, and operating procedures. The system design 
and individual module descriptions are also included. Var- 
ious appendixes contain the data base file formats and the 
system implementation procedures. This document was also 
issued as Computer Sciences Corporation document 
CSC/S D-7 9/610 3. FEM 
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2.17 EVALUATION OF THE CAINE, FARBER, AND GORDON PROGRAM 
DESIGN LANGUAGE (PDL) IN THE GODDARD SPACE FLIGHT 
CENW^nfG^FCr^CUDE~3¥0~EOFTWARE DESIGN ENVIRONMENT, 
SEL-79-004, C. E. GOOREVICH, A. L. GREEN, AND 
W. J. DECKER, SEPTEMBER 1979, 46 PAGES 

This document reports the results of a study of the useful- 
ness of program design languages (PDLs) for flight dynamics 
software development at GSFC. The following PDLs were ex- 
amined and compared: 

• Telemetry Computation Branch PDL 

• Linger and Mills PDL 

• Caine, Farber, and Gordon PDL 

The last PDL was selected for intensive study. Its advan- 
tages and disadvantages in the flight dynamics environment 
were evaluated. Appendixes include examples of the use of 
the Caine, Farber, and Gordon PDL and the processor output. 
This document was also issued as Computer Sciences Corpora- 
tion document CSC/TM-79/6263 . NSTF 
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2.18 PROCEEDINGS FROM THE FOURTH SUMMER SOFTWARE ENGINEERING 
WORKSHOP , SEL-79-005, NOVEMBER 1979, 282 PAGES 


This document reproduces the presentations made by partici- 
pants at the Fourth Summer Software Engineering Workshop 
held on November 19, 1979, at GSFC. The presentations were 
grouped into the following panels: 

• The Software Engineering Laboratory 

• Data collection 

• Experiments in methodology evaluation 

• Software resource models 

• Models and metrics of software development 

This fourth workshop focused on actual experiences of data 
collection and the application of software methodologies, 
models, and tools. Approximately 100 persons attended this 
meeting. NTIS 
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2. 19 FUNCTIONAL REQUIREMENTS/SPECIFICATIONS FOR CODE 580 

Configuration analysis tool (cat)", sel-so-ooi, 

F. K. BANKS, A. L. GREEN, AND C. E. GOOREVICH, 

FEBRUARY 1980, 49 PAGES 

This document presents the functional requirements and soft- 
ware specifications for a configuration management utility. 
The Configuration Analysis Tool (CAT) is an interactive pro- 
gram for recording and reporting the status of software de- 
velopment and maintenance activities. CAT was developed for 
use by flight dynamics software development projects. 

This document describes the hardware and operating system 
configuration, software system functions, and operational 
environment. The appendix contains data set formats. This 
document was also issued as Computer Sciences Corporation 
document CSC/TM-80/6051. FEM 
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2. 20 MULTI-LEVEL EXPRESSION DESIGN LANGUAGE - REQUIREMENT 
LEVEL (MEDL-R) SYSTEM EVALUATION , SEL-80-002, 

W. J. DECKER AND C. E, GOOREVICH, MAY 1980, 91 PAGES 

This document presents the results of an evaluation of the 
suitability of the Multi-Level Expression Design Language - 
Requirement Level (MEDL-R) for use in flight dynamics soft- 
ware development at GSFC. The evaluation team studied the 
MEDL-R concept of requirements languages, the functions per- 
formed by MEDL-R, and the MEDL-R language syntax. The docu- 
ment contains recommendations for changes to the MEDL-R 
system that would make it more useful in the flight dynamics 
environment. This document was also issued as Computer 
Sciences Corporation document CSC/TM-80/6093. NSTF 
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2 • 21 MULTI-MISSION modular spacecraft ground support soft - 
ware SYSTEM (MMS/GSSS) STATE-OF-THE-ART COMPU'ffEft 
SYSTEMS/COMPATIBILITY STUDY , SEL-80-003, T. WELDEN , 

M. MCCLELLAN, AND P. LIEBERTZ , MAY 1980, 66 PAGES 

This document describes the results of an evaluation of the 
compatibility between the ModComp IV/35 and the VAX-11/780 
computers with respect to a specific software system, the 
Multi-Mission Modular Spacecraft Ground Support Software \ 

System (MMS/GSSS) . The degree of compatibility watt measured' 

> 

by comparing the results of benchmark tests run on both sys- 
tems. The tests examined input/output services, FORTRAN 
language implementation, and execution timing. The compati- 
bility of peripheral devices and system command languages 
was considered in lesser detail. Significant incompatibil- 
ities were found in all areas examined. This document was 
also issued as Computer Sciences Corporation document 
CSC/TM- 8 0/6154 . NTIS 


2,22 CONFIGURATION ANALYSIS TOOL (CAT) SYSTEM DESCRIPTION 
AND USER’S GUIDE , SEL-80-ti04, F. K. BANKS - , 

Vf. J. DECKER^ "J. G. GARRAHAN, ET AL. , OCTOBER 1980, 

142 PAGES 

This document is the system description and user's guide for 
a configuration management utility. The Configuration Anal- 
ysis Tool (CAT) performs interactive recording and reporting 
of the status of software development and maintenance activ- 
ities. CAT enhances the ability of a manager to monitor and 
control a large software development project. 

This document provides a description of the system organiza- 
tion, operating procedures, source listings, file formats, 
error messages, and implementation instructions. This docu- 
ment was also issued as Computer Sciences Corporation docu- 
ment CSC/SD- 80/6089 . FEM 
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2, 23 A STUDY OF THE MUSA RELIABILITY MODEL # SEL-80-005, 

A , M. MILLER, NOVEMBER 1980, 94 PAGES 

This document describes a study in which the Musa reliabil- 
ity model was applied to three software projects developed 
for GSFC, with the goal of determining whether the model 
could be used in the flight dynamics environment as a soft- 
ware management tool. One purpose of the model is to pre- 
dict the total number of errors in a piece of software 
undergoing testing. Actual times between failures and their 
associated run times were fitted to the Musa equation in an 
iterative procedure. Of the three projects studied, the 
results fee one converged to a value 25 percent higher than 
the actual number of errors; the other two did not converge 
at all. 

The document discusses the assumptions underlying the model 
and evaluates the characteristics of the environment that 
could affect these assumptions. Suggestions are offered 
about changes that could be made in the environment to 
better meet the assumptions. This document was originally 
prepared as a Master's Thesis at the University of Maryland. 
NSTF 
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2.24 PROCEEDINGS FROM THE FIFTH ANNUAL SOFTWARE ENGINEERING 
WORKS H6P , SEE- 80 -06 6, NOVEMBER i98d7~T^PAGES 

This document reproduces the presentations made by partici- 
pants at the Fifth Annual Software Engineering Workshop held 
on November 24 , 1980, at GSFC. The presentations were 
grouped into the following panels: 

• The Software Engineering Laboratory 

• Software cost/resource modeling 

• Software reliability 

• Measurement of the development process 

This fifth workshop focused on actual experiences with the 
application of software methodologies and models. Approxi- 
mately 140 persons attended this meeting, NTIS 
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2.25 AN APPRAISAL OF SELECTED COST/RESOURCE ESTIMATION 

MODELS FOR SOFTWARE SYSTEMS , SEL-80-0Q7, J. F. COOK 
AND F. E. McGARRY , DECEMBER 1980, 41 PAGES 

This document presents the results of an evaluation and com 
parison of seven cost/resource estimation models based on 
SEL data. The following models were considered; 

• Doty 

• Walston/Felix 

• Tecolote 

• GRC 

• SLIM 

• PRICE S3 

• SEL Meta-Model 

The validity of the theoretical bases of these models was 
not analyzed. The objective of the appraisal was simply to 
determine how well SEL data conformed to the predictions of 
various models. This document was also issued as Goddard 
Space Flight Center document X-582-81-1. NTIS 
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2.26 MODELS AND METRICS FOR SOFTWARE MANAGEMENT AND 

ENGINEERING / SEL-80-008, V. R. BASILI, 1980, 349 PAGES 


This document is a tutorial on quantitative methods of soft- 
ware management and engineering. A quantitative methodology 
is needed to evaluate, control, and predict software devel- 
opment and maintenance costs. This quantitative approach 
allows cost, time, and quality tradeoffs to be made in a 
systematic manner. The tutorial focuses on numerical 
product-oriented measures such as size, complexity, and 
reliability and on resource-oriented measures such as cost, 
schedules, and resources. Twenty articles from software 
engineering literature are reprinted in this document. The 
articles are organized into the following sections: 

• Resource models 

• Changes and errors 

• Product metrics 

• Data collection 

Successful application of these techniques, however, re- 
quires a thorough knowledge of the project under development 
and any assumptions made. Only then can these techniques 
augment good managerial and engineering judgment. JAO 


This document was published as the IEEE tutorial. M odels and 
Metrics for Software Management and Engineering , New York: 
Computer Societies Press, 1980. 
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2.27 GUIDE TO DATA COLLECTION, SEL-81-101, V. E. CHURCH, 

’67 'nT'CaRT)'; AfoD " P. 6. McGARRY , AUGUST 1982, 123 PAGES 

This document presents guidelines and recommendations for 
collecting software development data. The guide describes 
the motivation, planning , implementation, and management of 
a data collection effort. Other topics covered include 
types, sources, and availability of data; methods and costs 
of data collection; types of analyses supported; and warn- 
ings a *3 suggestions based on SEL experience. The appen- 
dixes include facsimiles of SEL data collection forms and a 
glossary of software engineering terms. 

This document, abstracted and generalized from 5 years of 
SEL data collection experience, is intended to be a practi- 
cal guide for software managers and engineers. It was also 
issued as Computer Sciences Corporation document 
CSC/TM-82/6137. NTIS 


The previous version of this document was Guide to Data 
Collection , SEL-81-001, V. E. Church, D. N. Card, and 
PV E. McGa rry , September 1981. 
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2.28 SOFTWARE ENGINEERING LABORATORY (SEL) DATA BASE 
ORGANIZATION AND USER'S GUIDE, SEL-81-002, " 

tbt vrmmm ; tsttrsf; m t. e. mcgarry, 

SEPTEMBER 1981 f 171 PAGES 

This document describes the structure of the SEL data base. 
The data base contains three types of files s header, form 
data, and auxiliary files. The format and contents of each 
file are described in detail. File access and programming 
considerations are also discussed. The operation of some 
data base reporting software is reviewed* The appendixes 
include facsimiles of SEL data collection forms and a glos- 
sary of software engineering terms. This document was also 
issued as Computer Sciences Corporation document 
CSC/SD-81/6011UD1 . NSTF 


2.29 SOFTWARE ENGINEERING LABORATORY (SEL) DATA BASE MAINTE - 
NANCE SYSTEM (DBAM) "USER'S GUIDE AND SYSTEM DESCRIP- 
TION, SEL-81-003 , D. N. CARD, D. C. WYCKOFF, AND 
G. PAGE, SEPTEMBER 1981, 187 PAGES 

This document explains the organization and operation of the 
SEL Data Base Maintenance System (DBAM) . The system pro- 
vides the following functions: 

• Create new data base files 

• Archive data base on tape 

• Restore data base from tape 

• Update existing data base files 

The document also describes the use of these functions, pos- 
sible error messages, and operating restrictions. The ap- 
pendixes contain baseline diagrams, component descriptions, 
task-build instructions, and file definitions. This docu- 
ment was also issued as Computer Sciences Corporation docu- 
ment CSC/TM-81/6102 . FEM 
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2.30 THE SOFTWARE ENGINEERING LABORATORY , SEL-81-104, 

D. N. CARD, F. E. McGARRY, G. PAGE, ET AL. , 

FEBRUARY 1982, 121 PAGES 

This document describes the history, organization, opera- 
tion, and research results of the SEL. The SEL is a joint 
effort of GSFC, Computer Sciences Corporation, and the 
University of Maryland. The objective of the SEL is to 
study and improve the software development process in the 
GSFC environment. The SEL has conducted extensive research 
in the following areas of software engineering: 

• Methodology evaluation 

• Tool evaluation 

• Resource models 

• Reliability models 

• Software measures 

The document outlines SEL efforts in these areas and pre- 
sents some preliminary conclusions based on this work. The 
appendixes include descriptions of the software projects 
studied and summary statistics derived from this data. This 
document was also issued as CSC/TM-82/6033. NT IS 


The previous version of this document was The Software Engi 
neering Laboratory , SEL-81-004, D. N. Card, F. E. McGarry, 
G. Page, et al., September 1981. 
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2.31 RECOMMENDED APPROACH TO SOFTWARE DEVELOPMENT, 

SEL-81-105, S. ESLINGER , F. E. McGARRY , AND G. PAGE, 
MAY 1982, 135 PAGES 

This document presents recommendations for a disciplined 
approach to software development, based on data collected 
and studied by the SEL since 1977 for approximately 
40 flight dynamics software projects. It describes the 
major activities, end products, methodologies, tools, 
models, and measures applicable to each phase of the soft- 
ware life cycle. 

The emphasis of the document is on management considera- 
tions, including the activities specific to each phase of 
the life cycle and the general management concerns tran- 
scending phase boundaries, such as development planning and 
performance monitoring. Specific recommendations are made 
with respect to determining project quality. The appendix 
defines contents of requirements, design, system, and user 
documentation. This document was also issued as Computer 
Sciences Corporation document CSC/TM-82/6070 . NTIS 


The previous version of the document was Standard Approach 
to Software Development , SEL-81-005, V. E. Church, 

F. E. McGarry, and G. Page, September 1981. 
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2.32 SOFTWARE ENGINEERING LABORATORY (SEL) DOCUMENT LIBRARY 
(DOCHBj SYSTEM DESCRIPTION AND USER'S GUIDE, 

s-el'-bt- iJ(T<5 - ; ' 1 w r "■?r. - “'rAy lor ■■snc 1 wr j; decker; — 

DECEMBER 1981, 196 PAGES 

This document presents the system description and user's 
guide for the Document Library (DOCLIB) support software. 

The DOCLIB system consists of a set of programs that main- 
tain and access a computer data base containing descriptions 
of documents available from a library maintained by the SEL. 
The maintenance program, LIBMGR, is used to add, delete, or 
modify document descriptions. The access program, DOCLIB, 
is used to select and display specific descriptions. 

Documents can be selected by author, title fragment, refer- 
ence number, organization, or subject. This document de- 
scribes the system organization, operating procedures, file 
structures, and system generation. This document was also 
issued as Computer Sciences Corporation document 
CSC/S D- 81/ 6116 . FEM 


2.33 SOFTWARE ENGINEERING LABORATORY (SEL) COMPENDIUM OF 
TOOLS (REVISION 1) f SEL-81-107, W. J. DECKER, 

W. A. TAYLOR, AND E. J. SMITH, FEBRUARY 1982, 76 PAGES 

This document contains a series of brief descriptions of 
software tools available on the SEL computers (a PDP-11/70 
and a VAX-11/780) . The tools described in this document 
support the following applications: 

• Cost and resource modeling 

• Configuration management 

• Software library management 

• Data base maintenance 

• Document library indexing 

• Financial reporting 

• Requirements analysis 

• Source code analysis 

• Structured FORTRAN 

These brief descriptions help potential users to judge the 
suitability of these programs to their needs. This document 
was also issued as Computer Sciences Corporation document 
CSC/TM-82/6038. NTIS 


The previous version of this document was Software Engineer - 
ing Laboratory (SEL) Compendium of Tools , SEL-81-007, 

W. J. Decker, E. J. Smitn, W. A. Taylor, et al., February 
1982. 
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2.34 COST AND RELIABILITY ESTIMATION MODELS (CAREM) USER'S 
GUIDE # SEL-81-Q08 , J. F. COOK AND E. EDWARDS, 

FEBRUARY 1981, 28 PAGES 

This document describes the operation of an interactive 
software cost and reliability modeling utility. Tb'i Cost 
and Reliability Estimation Models (CAREM) program allows the 
user to fit any of several common models to a selected sub- 
set of SEL data. The following models are available in the 
program: 

• Doty 

• GRC 

• Tecolote 

• Walston/Felix 

This document includes a brief description of each model, 
operating instructions, and sample sessions. This document 
was also issued as a Goddard Space Flight Center technical 
memorandum. FEM 
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2 . 35 SOFTWARE ENGINEERING LABORATORY PROGRAMMER WORKBENCH 
PHASE 1 EVALUATION^ SEL-81-009, W. J. DECKER AND 
F. E. McGARRY, MARCH 1981, 29 PAGES 

This document summarizes an initial effort to develop a pro- 
grammer workbench for flight dynamics software development 
activities. Phase 1 of the programmer workbench consists of 
the design of three components: the communications link, 

the command language processor, and the software tools pack- 
age. The document also contains a brief description and 
evaluation of the design of each component. Some recommen- 
dations for future work are made. This document was also 
issued as Computer Sciences Corporation document 
CSC/TM-81/6091. NSTF 
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2.36 EVALUATION OF AN INDEPENDENT VERIFICATION AND VALIDA - 
TION TTV&Vy METHODOLOGY FOR FLIGHT"1)YNAMICS , 
SEL-8l-iib/ G. PAGE AND F. E.McGAKrY, DECEMBER 1982, 
132 PAGES 

This document describes an experiment in the application of 
an independent verification and validation (IV6V) methodol- 
ogy to the development of flight dynamics software at GSFC. 
IV&V is the systematic evaluation of computer software by an 
organization that is independent of the development organi- 
zation, JV&V is expected to provide earlier error detection 
and better quality control over the development process. 

This document describes the environment, staffing, and re- 
sults of the experiment. Costs and error rates are compared 
with those of similar projects developed without IV&V. An 
IV&V methodology is found to be appropriate for very large 
projects and for those with high reliability requirements. 
This document was also issued as Computer Sciences Corpora- 
tion document CSC/TM-82/6197. FEM 


The previous version of this document was Performance and 
Evaluation of an Independent Software Verification and Inte 
oration Process, SEL-81-010, G. Page and F. eT McGarry, May 

w er: 
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2.37 EVALUATING SOFTWARE DEVELOPMENT BY ANALYSIS OF CHANGE 

DATA, SEL-81-011, D. M. WEISS, NOVEMBER 1981/ 272 PAGES 


This document reports the results of an analysis of change 
data from five different software development projects in 
two different environments. A common data collection meth- 
odology was applied at both GSFC and the Naval Research Lab- 
oratory (NRL) . This document describes the data collection 
methodology employed, software projects studied, and the 
effects of changes on software development. 

The results of this study indicate that the data collection 
methodology is effective and easily extendable to new soft- 
ware development environments. Although the GSFC and NRL 
environments differed somewhat in their objectives and ap- 
proach to software development, the software produced by 
both groups was similar with respect to changes and errors. 
The results presented in this document include: (1) dis- 

tributions of causes of change, sources of errors, and dif- 
ficulty of finding errors and (2) tabulations of changes 
according to number of components changed, changes according 
to subsystem, difficulty of change (error) according to 
source of change (error) , and source of error according to 
programmer. This document was also issued as University of 
Maryland Technical Report TR-1120. FEM 
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2. 38 THE RAYLEIGH CURVE AS A MODEL FOR EFFORT DISTRIBUTION 
OVfiA THE LIFE OP MEDIUM SCALE SOFTWARE SYSTEMS, 
SEL-81-612, G. 0. PICASSO/ DECEMBER 1981, 133 PAGES 

This document discusses some of the factors affecting the 
accuracy of resource models applied to medium-scale software 
systems. Putnam has shown that the Rayleigh curve is an 
adequate model for the life-cycle effort distribution of 
large-scale systems. Previous investigations of the appli- 
cability of this model to medium-scale software development 
efforts have met with mixed results. The results of the 
earlier investigations are confirmed in this analysis. The 
reasons for the failure of the models are found in the sub- 
cycle (phase) effort data. There are four contributing fac- 
tors: uniqueness of the environment studied, the influence 

of holidays, varying management techniques, and differences 
in the data studied. This document was also issued as 
University of Maryland Technical Report TR-1186. FEM 
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2.39 PROCEEDINGS FROM THE SIXTH ANNUAL SOFTWARE ENGINEERING 

WOrKSHOR , sel-61-613, December 1961. 26?'pa"ges 


This document reproduces the presentations made by partici- 
pants at the Sixth Annual Software Engineering Workshop held 
on December 2, 1981, at GSFC. The presentations were 
grouped into the following panels: 

• Evaluating software development characteristics 

• Software metrics 

• Software models 

• Software methodologies 

This sixth workshop was an attempt to gather the experiences 
of software developers in applying modern programming prac- 
tices and other software engineering techniques. Approxi- 
mately 200 persons attended this meeting. NTIS 


2.40 AUTOMATED COLLECTION OF SOFTWARE ENGINEERING DATA IN 
THE SOFTWARE ENGINEERING LABORATORY (SEL) , SEL-81-014, 
A. L. GREEN, W. J. DECKER, AND F. E. McGARRY, 
SEPTEMBER 1981, 72 PAGES 

This document presents the results of an analysis of SEL 
data collection procedures. The principal questions 
addressed are what current manual procedures could be auto- 
mated and how these automated procedures could be in- 
corporated in the SEL data base system. The functional 
requirements of such a system are identified and explained. 
The automatable sources of data identified in this report 
include the following: 

• Computer accounting information 

• Requirements language tools 

• Program design language tools 

• Programmer workbench features 

• Source code analyzer program 

This document was also issued as Computer Sciences Corpora- 
tion document CSC/TM-81/6222. NSTF 
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2.41 EVALUATION OF MANAGEMENT MEASURES OF SOFTWARE DEVELOP - 
MENT , SEL-d2-( r 0i, G. "i'AGE, D. N. CARD , AND 
FTe. McGARRY , SEPTEMBER 1982, VOL. li 143 PAGES, 

VOL. 2: 379 PAGES 

This two-volume document reports the results of an evalua- 
tion of a large set of software development measures rele- 
vant to the GSFC environment. The purposes of the analysis 
were to characterize the current software development proc- 
ess in one environment by identifying important qualities 
and corresponding measures and to evaluate the effectiveness 
of specific tools and techniques in this environment. The 
measures studied were counts, ratios, and management- 
supplied ratings of various elements of the software devel- 
opment process. The measures are high level in that each 
describes some aspect of an entire software project (or a 
large part of it) rather than individual components of the 
project. 

Volume 1 explains a conceptual model of software develop- 
ment, the data classification scheme, and the analytic pro- 
cedures. Factor analysis, cluster analysis, and a test of 
normality were used. This volume summarizes the results of 
those analyses and recommends specific software measures for 
collection and monitoring. Volume 1 also reproduces in full 
the results of the computer analyses. 

Volume 2 presents a detailed description of the data ana- 
lyzed, including definitions of measures, lists of values, 
and Summary statistics. Although the information contained 
in Volume 2 was essential to the development of the explana- 
tion and summary presented in Volume, 1, it is not essential 
to the understanding of that explanation and summary. How- 
ever, Volume 2 is useful in its own right as a source of 
data and a reference for future research. This document was 
also issued as Computer Sciences Corporation document 
CSC/TM- 8 2/6063. FEM 
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2.42 FORTRAN STATIC SOURCE CODE ANALYZER PROGRAM (SAP) 
gYgggg DESCRIPTION # &EL-82-002# W. J. DECKER AND 
W. A. TAYL6 r, AUGUST 1982, 227 PAGES 

This document is the system description of the FORTRAN 
Static Source Code Analyzer Program (SAP) . SAP is a soft- 
ware tool that scans FORTRAN source code and tabulates 
occurrences of specific program features, such as executable 
lines of code, decisions, and input/output statements. 

This document defines the processing methods and the compo- 
nents of SAP. The appendixes fully describe the modules, 
COMMON blocks, and files. System generation procedures are 
also explained. Some important considerations for imple- 
menting SAP on computers other than the PDP-11/70 or the 
VAX-11/780 are discussed. This document was also issued as 
Computer Sciences Corporation document CSC/SD-82/6045. FEM 
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2.43 SOFTWARE ENGINEERING LABORATORY (SEL) DATA BASE REPORT - 
ING SOFTWARE USER 1 S GUIDE AND SYSTEM DESCRIPTION , 
SEL-82-00 3 , P. LO, SEPTEMBER IWTT 544 PAGES 

This document is the SEL data base reporting software user's 
guide and system description. The SEL data base reporting 
programs provide formatted listings and summary reports of 
the SEL data base contents. 

This document is intended to serve as a reference for the 
SEL data base administrator, librarians, and programmers and 
for managers and researchers using the SEL data base. It 
describes the operating procedures and system, information 
for 18 different reporting software programs and provides 
sample output reports obtained from each program. The rela- 
tionship of this software to other data base software is 
also discussed. This document was also issued as Computer 
Sciences Corporation document CSC/SD-82/6083. FEM 
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2 *44 COLLECTED SOFTWARE ENGINEERING PAPERS: VOLUME 1, 

SEL-82-004, JULY 1982, 118 PAGES 


This document is a collection of technical papers produced 
by participants in the SEL during the 5-year period ending 
December 31, 1981. The goal of the document is to make 
available, in one reference, some results of SEL research 
that originally appeared in a number of different forums. 
Although these papers cover a wide range of topics related 
to software engineering, they do not completely describe the 
activities and interests of the SEL. 

For the convenience of this presentation, the 10 papers are 
organized into 4 major topics: 

• SEL organization 

• Resource models 

• Software measures 

• Software engineering applications 

The SEL is still actively working to understand and improve 
the software development process at GSFC. Future efforts 
will be documented in subsequent volumes of the Collected 
Software Engineering Papers and in other SEL publications. 
FEM 


2-45 


SECTION 3 


- SELi-RELATED LITERATURE 


3-1 


8978 


3.1 BAILEY, J. W., AND V. R. BASILI, ”A META-MODEL FOR SOFT- 
WARE DEVELOPMENT RESOURCE EXPENDITURES,” PROCEEDINGS OF 
THE FIFTH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEER- 
W?T. NEW YORK: COMPUTER SOCIETIES PRtssfr, 1981, 

T0~" PAGES 

This technical paper describes an effort to produce a model 
of software development resource expenditures that can be 
generalized to a number of situations. Many models have 
been proposed over the last several years. However, experi- 
ence has shown that differences in the data collected, types 
of projects developed, and environmental factors limit the 
transportability of these models from one organization to 
another. This conclusion is reasonable because a model de- 
veloped in any given environment will reflect only the 
impact of factors that have a variable effect in that envi- 
ronment. Factors that are constant in that environment (and 
therefore do not affect productivity) may have different or 
variable effects in another environment. 

This paper describes a model-generation process that permits 
the development of a resource estimation model for any par- 
ticular organization. The process provides the capability 
to produce a model that is tailored to the organization and 
can be expected to be more effective than any model orig- 
inally developed for another environment. The model is 
demonstrated here using data collected by the SEL at GSFC. 

JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers: Volume 1 , July 1982. 
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3.2 BANKS, F. K., "CONFIGURATION ANALYSIS TOOL DESIGN," 
COMPUTER SCIENCES CORPORATION, TECHNICAL MEMORANDUM, 
MARCH 1980, 99 PAGES 

This technical memorandum presents the design of a Configu- 
ration Analysis Tool (CAT) . CAT is an interactive program 
that maintains a file of components under development and 
produces reports of software development activity. This 
memorandum includes the baseline diagram, system descrip- 
tion, and detailed module descriptions and defines menus, 
displays, and file formats. 
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3.3 BASILI , V. R., "MODELS AND METRICS FOR SOFTWARE MANAGE 
MENT AND ENGINEERING," ASME ADVANCES IN COMPUTER TECH- 
NOLOGY , JANUARY 1980, VOL. 1, 12 PAGES 

This technical paper attempts to characterize several quan- 
titative models and measures of the software development 
process. These models and measures deal with various as- 
pects of the software process and product, including re- 
source estimation, complexity, reliability, and size. The 
relationship of these models and measures to the software 
development life cycle is also discussed. Finally, the ex- 
tent to which the various models have been applied in pro- 
duction environments and the success they have achieved is 
indicated. JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers: Volume 1 , July 1982. 
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3.4 BASILI, V. R, , "SOFTWARE ENGINEERING LABORATORY RELA- 
TIONSHIPS FOR PROGRAMMING MEASUREMENT AND ESTIMATION, " 
UNIVERSITY OF MARYLAND, TECHNICAL MEMORANDUM, OCTOBER 
1979, 24 PAGES 

This technical memorandum reproduces a series of graphs 
showing the relationships among several software measures. 
The graphs were produced by regression analyses of SEL data. 
The relationships illustrated are those investigated by 
Walston and Felix. See also Section 3.6. 


3.5 BASILI, V.R., AND J. BEANE, "CAN THE PARR CURVE HELP 
WITH MANPOWER DISTRIBUTION AND RESOURCE ESTIMATION 
PROBLEMS?", JOURNAL OF SYSTEMS AND SOFTWARE, FEBRUARY 
1981, VOL. 2, NO. 17 11 PA<SeS 

This technical paper analyzes the resource utilization model 
developed by Parr. The curve predicted by the model is com- 
pared with several other curves, including the Rayleigh 
curve, a parabola, and a trapezoid, with respect to how well 
they fit manpower utilization. The evaluation is performed 
for several flight dynamics projects of the 6- to 12-man-year 
effort range that were studied by the SEL. 

The conclusion drawn is that the Parr curve can be made to 
fit the data better than the other curves. However, because 
of the noise in the data, it is difficult to confirm the 
shape of the manpower distribution from the data alone and 
therefore difficult to validate any particular model. More- 
over, since the parameters used in the curve are not easily 
calculable or estimable from known data, the curve is not 
effective for resource estimation. JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers: Volume 1 , July 1982. 
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3.6 BASIL! , V. R., AND K, FREBURGER, "PROGRAMMING MEASURE- 
MENT AND ESTIMATION IN THE SOFTWARE ENGINEERING LABO- 
RATORY," JOURNAL OF SYSTEMS AND SOFTWARE, FEBRUARY 1981, 
VOL. 2, Nd). 17 ll pa<Ses 

This technical paper presents an examination of a set of 
basic relationships among various software development meas- 
ures, including size, effort, project duration, staff size, 
and productivity. Correlations among these measures are 
computed. The data used comes from 15 flight dynamics soft- 
ware development projects studied by the SEL. Certain rela- 
tionships are derived in the form of equations, and these 
equations are compared with a set derived by Walston and 
Felix for IBM Federal Systems Division project data. 
Logarithmic transformations were performed on the data for 
some analyses. Although the equations do not have the same 
coefficients, they are seen to have similar exponents* In 
fact, the SEL-derived equations tend to be within one stand- 
ard error of the estimates provided in the IBM equations. 

JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers; Volume 1 , July 1982. 
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3.7 BASILI, V. R, , AND B, T. PERRICONE, SOFTWARE ERRORS AND 
COMPLEXITY: AN EMPIRICAL INVESTIGATION , UNIVERSITY Ofr ~ 

MARYLAND , TECHNICAL REPORT TR-1195, AUGUST 1982, 

29 PAGES 

This document reports the results of an analysis of error 
data obtained from ® flight dynamics software project 
studied by the SEL. The distributions of errors by type and 
location are identified and discussed. Correlations among 

module size, complexity, and error rate are then described 

« 

and evaluated. Modified and new modules are shown to have 
similar error characteristics. An alternative error classi- 
fication scheme is developed. Finally, an attempt is made 
to compare these results with those of other researchers in 
the field. * 
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3.8 BASILI, V. R., AND T. PHILLIPS , "EVALUATING AND COM- 
PARING SOFTWARE METRICS IN THE SOFTWARE ENGINEERING 
LABORATORY , " PROCEEDINGS OF THE ACM SIGMETRICS 
SYMPOSIUM/WORKSHOP: QUALITY METRICS , MARCH 1981, 

TTpHSES 

This technical paper describes an effort to identify the 
best measures of software development effort and software 
complexity. Four software projects studied by the SEL pro- 
vide the data for the analysis. The data is screened to 
ensure its validity. Next, estimating equations are derived 
for effort and errors using the various measures studied in 
the analysis. Correlations are shown to increase as the 
reliability of the data increases due to screening. Thus, a 
procedure is demonstrated for removing noise from the data 
and making possible meaningful comparisons of software 
metrics. JAO 


Th i s t ech n real pape r also appears in SEL-82-004* Collected 
Software Engineering Papers; Volume 1 , July 1982. 
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3.9 BASILI, V. R. , R. W. SELBY, AND T. PHILLIPS, METRIC 

ANALYSIS AND DATA VALIDATION ACROSS FORTRAN PROJECTS , 
UNIVERSITY OF MARYLAND, TECHNICAL REPORT, NOVEMBER 1982 

This technical paper reports the results of an analysis of 
the relationship of Halstead measures, McCabe complexity 
measures, and other software measures to software develop- 
ment effort and errors, Effort is defined in terms of 
staff-hours from the establishing of functional specifica- 
tions through acceptance testing. Errors are counted dis- 
cretely and weighed according to effort to correct. The 
data studied was collected by the SEL in a production en- 
vironment. Cross-checks of the data indicated a need for 
large-scale data validation. The strongest correlations 
were obtained when the modules of individual programmers 
were considered independently. However, neither Halstead's 
effort measure, McCabe's eyelorasfeie complexity measure, nor 
lines of source code was convincingly more accurate as an 
estimator than the others. 
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3.10 BASILI, V. R. , AND R. REITER, "EVALUATING AUTOMATABLE 
MEASURES FOR SOFTWARE DEVELOPMENT," PROCEEDINGS OF THE 
WORKSHOP ON QUANTITATIVE SOFTWARE MODELS FOR RELIABIL- 
ITY, COMPLEXITY, AND COST . NEW YORK: COMPUTER 

SOCIETIES PRESS, 1979, 10 PAGES 

This technical paper describes an approach to developing and 
evaluating automatable software measures u The experience of 
the SEL has shown that software data collection is an expen- 
sive activity that can significantly affect the software 
development process. Costs and effects can be minimized and 
data quality can be improved by automating the collection of 
measures wherever possible. 

This paper presents a set of automatable measures that were 
implemented and evaluated in a controlled experiment. The 
measures include computer job steps, program changes, pro- 
gram size, and software complexity. The results of the ex- 
periment indicate that the automated collection of these 
measures can be implemented effectively in production envi- 
ronments. JAO 


3-11 


3.11 BASILI, V. R. , AND M. V. ZELKOWXTZ, "DESIGNING A SOFT- 
WARE MEASUREMENT EXPERIMENT," PROCEEDINGS OF THE SOFT- 
WARE LIFE CYCLE MANAGEMENT WORKSHOP, SEPTEMBER "l 9 77 , 

I3TW5ES " 

This technical paper explains the research approach employed 
by the SEL to study the development of actual software de- 
velopment projects. The following types of experiments are 
performed by the SEL: 

• Screening 

• Semicontrolled 

• Controlled 

This paper discusses these experimental designs, potential 
confounding effects, and the statistical techniques used bo 
evaluate results. The effects on software developers of 
both learning during the experiment and an awareness of the 
experimental process itself are examined in detail. Fully 
controlled experiments are especially difficult to implement 
in a production environment, but sufficient control is pos- 
sible to evaluate the effects of software development meth- 
odologies. JAO 
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3.12 BASILI, V. R, , AND M. V. ZELK0WIT2 , "OPERATION OP THE 
SOFTWARE ENGINEERING LABORATORY, ” PROCEEDINGS OF THE 
SECOND SOFTWARE LIFE CYCLE MANAGEMENT WORKSHOP , 

AUGUST 1976 , "4 PAGES 

This technical paper describes the operation of the SEL. 
Software engineering data is regularly collected by the SEL 
from flight dynamics software development projects at GSFC. 
The assembled data supports an extensive program of software 
engineering research. This report also reviews SEL data 
collection and data processing activities and their rela- 
tionship to the research program. It also summarizes some 
ongoing resource estimation and error analysis research 
projects. JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers; Volume 1 , July 1982 
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3.13 BASILI, V. R. , AND M. V. ZELKOWITZ, "MEASURING SOFTWARE 
DEVELOPMENT CHARACTERISTICS IN THE LOCAL ENVIRONMENT, M 
COMPUTERS AND STRUCTURES , AUGUST 1978, VOL. 10, 5 PAGES 

This technical paper discusses the role of data collection 
in forecasting and monitoring software development projects 
in a production environment. The specific procedures of the 
SEL are reviewed, and SEL data collection forms are de- 
scribed. The paper also gives some examples of analyses 
that can be performed to support managing, understanding, 
and characterizing software development. The sample anal- 
yses identify specific measures for collection. JAQ 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers; Volume 1 , July 1982. 
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3.14 BASILI , V. R. , AND M. V. ZELK0WIT2 , "ANALYZING MEDIUM 
SCALE SOFTWARE DEVELOPMENT," PROCEEDINGS OF THE THIRD 
INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING. 

NEW YORK; COMPUTER SOCIETIES PRESS', MAV 1979, 8 PAGES 

This technical paper surveys SEL research activities in 
software engineering. The collection and analysis of data 
from software development projects is necessary for the de- 
finitive evaluation of software engineering methodologies 
and techniques. This paper describes the structure of the 
SEL and some of the early projects that were monitored. It 
also discusses the application of this data to resource 
utilization models and reliability studies. The principal 
contribution of the SEL, as reported in the paper, is the 
establishment of a facility for collecting the detailed data 
necessary for these analyses. JAO 
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3.15 BASILI, V. R., AND M. V. ZELKOWITZ, "THE SOFTWARE 

ENGINEERING LABORATORY: OBJECTIVES," PROCEEDINGS OF 

THE FIFTEENTH ANNUAL CONFERENCE ON COMPUTER PERSONNEL 
RESEARCH , AUGUST 1977, 14 PAGElST' 

This technical paper provides an overview of the SEL and its 
objectives. The original motivations for establishing the 
SEL were the high cost of software development and the sub- 
sequent need to optimize the development process. This 
paper discusses the following aspects of the SEL with re- 
spect to these motivations: 

• Specific objectives of the SEL 

• Software development factors to be investigated 

• Data collection techniques 

• Early SEL research activities 

The importance of defining consistent software development 
measures is a recurrent theme throughout the discussion. JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers: Volume 1 , July 1982. 
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3.16 CARD, D. N. , "EARLY ESTIMATION OP RESOURCES EXPEND- 
ITURES AND PROGRAM SIZE," COMPUTER SCIENCES CORPORA- 
TION, TECHNICAL MEMORANDUM, JUNE 1982, 24 PAGES 

This technical memorandum evaluates the suitability of sev- 
eral software measures as estimators of resource expendi- 
tures and program size early in the software life cycle. 

The estimating equation based on the most commonly employed 
measure, lines of source code, is explained and its limita- 
tions are identified. Several alternative measures are in- 
vestigated and found to give good results. The memorandum 
also includes computer-generated output of the least-squares 
regression analyses upon which the conclusions are based. 

FEM 
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3^17 CARD, D. N., AND M. G. ROHLEDER, "REPORT OF DATA 

EXPANSION EFFORTS," COMPUTER SCIENCES CORPORATION, 
TECHNICAL MEMORANDUM, SEPTEMBER 1982, 11 PAGES 

This technical memorandum describes some efforts made to 
expand the scope of data collected by the SEL. The SEL cur- 
rently collects data from the development of one specific 
type of software application. Ultimately, the SEL plans to 
collect data from other application types and from the main- 
tenance phase of software projects. This memorandum de- 
scribes the recent efforts of SEL personnel to include these 
additional data items in the SEL data base by tapping three 
new data sources within GSFC. FEM 
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3.18 CHEN , E. , AND M. V. ZELKOWITZ, "USE OF CLUSTER ANALYSIS 
TO EVALUATE SOFTWARE ENGINEERING METHODOLOGIES," 
PROCEEDINGS OF THE FIFTH INTERNATIONAL CONFERENCE ON 
SOFTWARE ENGINEERING ^ NEW YORK: COMPUTER SOCIETIES 

PRESS, 1981, 7 PAGES 

This technical paper describes an attempt to identify the 
characteristic effects of various methodologies on software 
development. Data collected by the SEL from five software 
projects was studied. Several objective measures were de- 
rived from the data, and their relationships to methodology 
use were studied with cluster analysis techniques. The 
analysis showed that the measures reflected the effects of 
methodologies on software development. JAO 


This technical paper also appears in SEL-82-004, Collected 
Software Engineering Papers; Volume l f July 1982. 


3-19 


8978 


3.19 FREBURGER, K., "A MODEL OF THE SOFTWARE LIFE CYCLE" 
(PAPER PREPARED FOR THE UNIVERSITY OF MARYLAND, 
DECEMBER 1978) , 27 PAGES 

This paper describes the software life cycle and its rela- 
tionship to the Rayleigh curve model of resource expendi- 
tures. The software life cycle is defined as a sequence of 
phases or steps through which a software system progresses 
until development is completed. Resource expenditures dur- 
ing the software life cycle can be modeled with a Rayleigh 
curve. This model allows staffing levels, development time, 
and total cost of development to be estimated. 


3.20 HISLOP, G,, "SOME TESTS OF HALSTEAD MEASURES" 

(PAPER PREPARED FOR THE UNIVERSITY OF MARYLAND, 
DECEMBER 1978) , 26 PAGES 

This paper attempts to evaluate the "software science" meas- 
ures devised by Halstead. It first explains the concepts of 
operators and operands and then defines some complex meas- 
ures composed of operators and operands. The predicted 
values of these measures are correlated with actual values 
for sample programs. The results of this analysis indicate 
that Halstead measures are relevant to some aspects of soft- 
ware development but not to all. 
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3.21 LANGE, S. F., "A CHILD’S GARDEN OF COMPLEXITY MEASURES” 
(PAPER PREPARED FOR THE UNIVERSITY OF MARYLAND, 

DECEMBER 1978) , 31 PAGES 

This paper investigates several forms of the McCabe measure 
of program complexity. The McCabe measure is defined, its 
relationship to structured programming is examined, and the 
McCabe concept itself is evaluated by comparison with data 
obtained from actual FORTRAN programs. Programmer-rated 
difficulty is found to be highly correlated with all forms 
of the McCabe complexity measure considered in this anal- 
ysis. Furthermore, these results support the proposed rela- 
tionship between structured programming and the various 
forms of the McCabe complexity measure. 
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3.22 MILLER, A. M., "A SURVEY OP SEVERAL RELIABILITY MODELS" 
(PAPER PREPARED FOR THE UNIVERSITY OF MARYLAND, 

DECEMBER 1978) , 19 PAGES 

This paper describes and compares three prominent software 
reliability models. It outlines the assumptions, limita- 
tions, and mathematical formulations of these models. The 
models studied are those developed by the following re- 
searchers : 

• M. L. Shooman 

• J. D. Musa 

• B.- Littlewood 

The predictions of the models are compared with the results 
obtained from an actual software project studied by the 
SEL. Firm conclusions cannot be drawn because of the in- 
adequacy of the data. However, recommendations are made for 
changes in the data collection process to facilitate this 
type of analysis. 


3.23 NATIONAL AERONAUTICS AND SPACE ADMINISTRATION, 

NASA SOFTWARE RESEARCH AND TECHNOLOGY WORKSHOP 
'(PROCEEDINGS) , MARCH i960, 139 PAGES 

This document reproduces the presentations made by partici- 
pants at the NASA Software Research and Technology Workshop 
held March 11 through 13, 1980, at NASA Headquarters in 
Washington, D. c. The overall intent of the workshop was to 
provide the basis for a coordinated agency-wide software 
research and technology program to meet agency requirements 
more efficiently and effectively. The topics of the work- 
shop were 

• Methodologies, tools, and techniques 

® Programming languages 

• Data base management software 

SEL data collection methods were recommended to provide the 
quantitative information necessary for evaluating entries in 
the three areas above. Approximately 45 persons attended 
the workshop. FEM 
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3.24 PAGE, G., "SOFTWARE ENGINEERING COURSE EVALUATION," 
COMPUTER SCIENCES CORPORATION, TECHNICAL MEMORANDUM, 
DECEMBER 1977, 41 PAGES 

This technical memorandum reproduces the results of an eval- 
uation by students participating in a software engineering 
methods course. The students were actively working on soft- 
ware developmen projects at the time the course was offered. 
The objective of the evaluation was to determine the 
students' reaction to, previous familiarity with, and subse- 
quent use of software engineering techniques. The memo- 
randum includes the evaluation forms completed by the 
students and a tabulation of the responses. FEM 
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3.25 PARR, F., AND D* M. WEISS, "CONCEPTS USED IN THE CHANGE 
REPORT FORM," NASA, GODDARD SPACE FLIGHT CENTER, 
TECHNICAL MEMORANDUM, MAY 1978, 9 PAGES 

This technical memorandum discusses the software engineering 
basis of the Change Report Form devised by the SEL* This 
form is used to collect change and error data from flight 
dynamics software development projects at GSFC. The memo- 
randum describes the questions that appear on the form and 
explains the possible answers. These questions attempt to 
determine the nature of the change/error, the cause of the 
change/error, the cost of the change/error, and other re- 
lated qualities. FEM 
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3.26 REITER, R. W., "THE NATURE, ORGANIZATION, MEASUREMENT, 
AND MANAGEMENT OP SOFTWARE COMPLEXITY" (PAPER PREPARED 
FOR THE UNIVERSITY OF MARYLAND, DECEMBER 1976) , 

47 PAGES 

This paper is a discussion of the relationship of software 
complexity to software development and maintenance. The 
sources, or factors, of software complexity are also con- 
sidered as well as complexity measures. The complexity of a 
software system is closely related to the difficulty of 
understanding that system and the way in which its elements 
fit and function together. The term "complexity," as de- 
fined in this paper, does not specifically refer to the dif- 
ficulty of algorithms. 
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SCHEFFER, P. A., AND C. E. VELEZ, "GSFC NAVPAK DESIGN 
HIGHER ORDER LANGUAGES STUDY: ADDENDUM, " MARTIN 

MARIETTA CORPORATION, TECHNICAL MEMORANDUM, 

SEPTEMBER 1977, 170 PAGES 

This technical memorandum reproduces the computer output on 
which the software design languages evaluation described in 
Section 2.5 was based in part. The materials reproduced 
include the Problem Statement Language (PSL) input (or 
source) and the following types of output produced by the 
PSL processor: 

• Formatted problem statement 

• Cross-reference index 

• Data-process report 

• Consists matrix 

• Extended picture report 

These tables and reports can be used to identify errors and 
ambiguities in the PSL representation of the software design. 
FEM 
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3.28 TURNER, C. , AND G. CARON, A COMPARISON OP RADC AND 
NASA/SEL SOFTWARE DEVELOPMENT DATA,' DATA & ANALYSIS 
CENTER FOR SOFTWARE, MAY 1981, $1 PAGES 

This document reports the results of an analysis of the re- 
lationship between project size and several other software 
measures. These measures include productivity, effort, du- 
ration, errors, and error rate. The analysis used data from 
two sources: Rome Air Development Center (RADC) and the 

SEL. Least-squares regression techniques were applied to 
both sets of data. Results obtained from the two sets of 
data were comparable. The conclusion cited in the report is 
that RADC and SEL data can be combined, in most cases, to 
obtain a larger sample without undesirable side effects. 
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3.29 TURNER y C. , G. CARON , AND G. BREMENT, NASA/SEL DATA 
COMPENDIUM , DATA & ANALYSIS CENTER FOR SOFTWARE, 

SPEC I AC PU'BL IC AT ION , MAY 1981, 80 PAGES 

This document summarizes the software engineering data col- 
lected by the SEL from flight dynamics development projects. 
A series of charts and graphs is presented for each project. 
These materials provide profiles of the development histo- 
ries of the projects and define some of the major character- 
istics of the software development process at GSFC. However, 
this document is a factual presentation rather than an 
analysis of the data. 
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3.30 WEISS, D. M., "ERROR AND CHANGE ANALYSIS," NAVAL 
RESEARCH LABORATORY, TECHNICAL MEMORANDUM, 

JULY 1979, 56 PAGES 

This technical memorandum discusses the essential concepts 
upon which any analysis of error and change data must be 
based. These concepts are the motivation for error/change 
analyses, the generation of hypotheses, and data collection 
procedures. The measures and distributions relevant to 
these analyses are defined and ranked according to their 
importance and the difficulty with which they are obtained. 
The memorandum also explains the derivations of these meas- 
ures and provides the outlines of software functions for 
calculating them. In short, this memorandum provides useful 
information for planning an analysis of error/change data. 
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3.31 WILLIAMSON, I. M. , “RESOURCE MODEL TESTING AND INFORMA- 
TION, " NAVAL RESEARCH LAORATORY, TECHNICAL MEMORANDUM, 
JULY 1979, 41 PAGES 

This technical memorandum reviews seven published resource 
estimation models. It identifies the parameters, explains 
the algorithm, and gives an example of application for each 
model. The models studied are 

• NARDAC 

• Doty 

• Tecolote 

• Aron 

• Walston/Felix 

• GRC 

• Wolverton 

Limitations and problems encountered in the application ex- 
amples or published presentations of these models are dis- 
cussed. 
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3.32 ZELKOWITZ, M. V., ’’RESOURCE ESTIMATION FOR MEDIUM 

SCALE SOFTWARE PROJECTS , ” PROCEEDINGS OF THE TWELFTH 
CONFERENCE ON THE INTERFACE OF STATISTICS AND COMPUTER 
SCIENCE . NEW YORK: COMPUTER SOCIETIES PRESS, 1979, 

6 PAGES 

This technical paper describes the analysis of resource 
estimation techniques that is being performed by the SEL. 

The data used in the analysis is collected from medium-scale 
flight dynamics software development projects at GSFC. A 
procedure to forecast accurately the cost and development 
time of these projects would be a valuable management tool 
in this environment. This paper documents a specific at- 
tempt to verify the resource estimation model based on the 
Rayleigh curve that was developed by Norden and Putnam. JAO 


This technical paper also appears in SEL-82-0Q4, Collected 
Software Engineering Papers; Volume 1 , July 1982. 
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3.33 ZELKOWITZ, M. V. , "DATA COLLECTION AND EVALUATION FOR 
EXPERIMENTAL COMPUTER SCIENCE RESEARCH," EMPIRICAL 
FOUNDATIONS FOR COMPUTER AND INFORMATION SCIENCE 
(PROCEEDINGS) , NOVEMBER 1982, 15 PAGES 

This technical paper reviews the data collection procedures 
of the SEL and shows how they are used to generate data with 
one particular software engineering experiment. The SEL 
collects process and product measures from actual scientific 
software projects using a combination of automated tools and 
questionaires. A project currently in progress, from which 
data is being collected, is a software prototyping effort. 
The goal of this experiment is to determine the costs and 
benefits of developing a prototype before developing a full 
system. JAO 
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3.34 ZELKOWITZ, M. V. , AND V. R. BASILI, ’’OPERATIONAL 
ASPECTS OF A SOFTWARE MEASUREMENT FACILITY, M 
PROCEEDINGS OF THE SOFTWARE LIFE CYCLE MANAGEMENT 

^M5^ r5fe'P l TlMfe'fiR"l$77; 11 pages 

This technical paper describes the data collection opera- 
tions of the SEL. The source of the data is a group of 
flight dynamics software development projects at GSFC. The 
paper describes seven data collection forms used by the 
SEL. The procedure for transferring data from the forms to 
the computer data base is outlined. Some of the validity 
checks performed on the data are identified. This procedure 
produces valid, relevant data with which significant soft- 
ware engineering research can be conducted. JAO 
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